1. Field of the Invention
The present invention relates generally to methods (and corresponding apparatus) for planning and executing the testing of computer systems. Such methods are generally referred to hereinafter as test planning and execution "models". More particularly, the invention relates to test planning and execution models that include methods (and corresponding apparatus) for generating a set of non-redundant (unique) test modules which define the steps needed to test a component or unit within a given computer system.
According to one aspect of the invention, the test planning and execution models contemplated herein may be characterized as processes in which a knowledge base, that includes a set of unique test modules, is first built and then utilized to generate non-redundant test cases.
The non-redundant test cases are generated by querying the constructed knowledge base for test modules fitting a given criteria, such as the component or unit under test, purpose of the test being run, etc. The non-redundant test cases may then be executed with the results serving as further input, according to a preferred embodiment of the invention, for updating the knowledge base.
Further aspects of the invention include methods (and corresponding apparatus) for actually generating the aforementioned set of non-redundant test modules; and for actually generating and executing the aforementioned non-redundant test cases.
The invention facilitates automatic redundancy checking of test modules used for testing a computer system utilizing predefined criteria expressed as a set of parameters aimed at detecting redundant tests. The invention also provides for the dynamic generation of test plans using the predefined criteria and non-redundant test modules.
2. Definitions
The following definitions are used herein:
Test Tool: A control program and all the needed hardware and software applied to test the functions of a computer system. Test tools may execute test cases on an individual basis (one at a time), or on a continuous basis.
Test Case: A program that is executed by a test tool and which has a specific test objective. Such a program executes a set of instructions and determines results, outputting the status of each unit or component under test.
Test Module: A knowledge base entry (or data set) that includes a set of parameters defining the steps needed to test a component or unit within a given computer system. A test module although being a data set is referred to herein as being "executed" when the aforementioned steps have been performed.
Cluster of Modules: A set of test modules exhibiting predefined characteristics based on given criteria. Also referred to in the art as a "Test Bucket".
Knowledge Base: A data base that is updated over time (on either a periodic or continuous basis) so that the data base as updated becomes an acquired knowledge about a given subject matter. For the purposes of the present invention, the data base becomes a knowledge base on how to test a computer.
"Investigation": Based on the input of a specified test module, the test planning and execution model contemplated by the invention outputs a number of test modules that having at least a predefined subset of parameters that are identical in value (as explained in detail hereinafter). By "investigation", as used herein, a user manually (or automatically by appropriate programing of a data processing device) reviews the modules output by the test planing and execution module and decides whether they are sufficient collectively or individually to accomplish a given test objective.
"Applying" a Knowledge Base: Once the knowledge base is established, a user prompts (queries) the knowledge base for modules to perform a specific test. The outcome of this prompt is a number of test modules that fit the prompt criteria. For example, a prompt may be:
1) Objective: Functional test
2) Unit: I/O
3) Component: Direct Access Storage Device (DASD)
The output of the prompt are the modules that fit the entered data (criteria) and this is referred to herein as "applying" a knowledge base.
3. Brief Description of the Prior Art
It is well known in the prior art to utilize knowledge bases for computer testing and fault diagnostic processing purposes. Examples of such systems include:
U.S. Pat. No. 4,985,857 which discloses a method and apparatus for diagnosing machines by using three unique modules: a Machine Information Database (MID), a Sensory Input Database (SID), and a Knowledge Base (KB);
U.S. Pat. No. 4,964,125 which discloses a method and apparatus for diagnosing equipment faults by utilizing an expert system including a knowledge base containing information regarding a module of the equipment;
U.S. Pat. No. 5,067,129 which discloses a service processor tester which automatically generates test cases by utilizing a knowledge base;
U.S. Pat. No. 4,847,795 which discloses a system for diagnosing defects in electronic assemblies by utilizing a knowledge base for storing information regarding the electronic assembly; and
U.S. Pat. No. 4,649,515 which discloses a generic facility for the construction of knowledge based expert systems for sensor and interactive based process diagnosis.
Still further examples of patents which illustrate the present state of the art of computer test equipment and fault diagnosis processes which utilize knowledge bases or expert systems include, U.S. Pat. No. 4,841,456; U.S. Pat. No. 4,972,453; U.S. Pat. No. 4,654,852; U.S. Pat. No. 4,914,924; and U.S. Pat. No. 4,881,230.
None of the aforementioned illustrative prior art references teaches, claims or even suggests a test planning and execution model (as defined herein) which is used to generate non-redundant tests which may be used to test a computer system based on user selected criteria input to the model.
In fact, the normal practice for planning and executing tests to exercise computer hardware and/or software is to manually define what testing is required and then hold "walk thrus" of the various defined test modules to be sure all areas to be tested are covered.
Accordingly, it would be desirable to provide (a) methods (and corresponding apparatus) for automating the test planning and execution process; (b) test planning and execution models that are specifically well suited for automatically detecting test module redundancy; (c) methods (and corresponding apparatus) which automatically provide a user with a set of appropriate candidate non-redundant test cases based on user selected criteria input to the model; and (d) test planing and execution models that can be rerun dynamically as a system design changes.
To complete the brief description of the prior art, an example of how the test planning and execution function is often carried out to test a computer system (without the benefit of the present invention), will be presented for the sake of illustration and comparison with the teachings of the invention set forth hereinafter. In particular, the example is based on the type of process used to test the IBM 9370 and IBM 9221 computer systems during their development phases ("IBM" is a registered trademark of the International Business Machines Corporation).
According to the exemplary prior art test planning and execution process the subject computer systems were first sub-divided into a number of components and units.
A unit, typically including a number of components, was defined (according to the exemplary process) as a functioning element within the microcode, or hardware, sub-system of the aforementioned computer systems under test. Examples of a unit within the microcode sub-system would be the processing unit (PU), the Support Processor (SP), the I/O Processor, etc. An example of a component within the Support Processor unit would be the Initial Microcode Load (IML).
After a given computer system was sub-divided as indicated hereinbefore, the responsibilities of testing the units and components was (as is often the case) distributed among teams within a system test organization. Each team proceeded by defining the scope and objective of the tests to be performed based on the system's specifications.
As a result, plans defining the steps needed to execute a test were generated. These plans were usually reviewed by other teams within the system test organization, and other development groups (e.g., Product Assurance, etc.) to assure consistent and complete tests were developed.
The main topic for discussion during such reviews was generally the coverage of the proposed plans. For example, a plan may have over-looked the testing of a particular component, or contained insufficient testing of another component. During these reviews, detailed discussions about similarities among test plans were often found to be difficult to conduct because they require the participants to learn about the scope, objective and strategy of all test plans generated by a test organization.
To this end, it would be desirable to reduce redundant testing within a test plan or among multiple test plans within a test organization, by establishing a common knowledge base which is applied to generate non-redundant test modules.
The common knowledge base may (after its construction) be used to generate non-redundant test modules. The models contemplated by the invention do not propose to replace the process of assigning teams to test the units within the system, and reviewing of test plans. Also, the model does not provide the capability to generate the most effective test per se, which is left to the experienced test teams. However, the proposed model compliments the processes of establishing and reviewing of test plans by reducing redundant testing.